A Generalized Fundamental Matrix for Computing Fundamental Quantities of Markov Systems

As is well known, the fundamental matrix (I − P + eπ) plays an important role in the performance analysis of Markov systems, where P is the transition probability matrix, e is the column vector of ones, and π is the row vector of the steady state distribution. It is used to compute the performance potential (relative value function) of Markov decision processes under the average criterion, such as g = (I −P + eπ)f where g is the column vector of performance potentials and f is the column vector of reward functions. However, we need to pre-compute π before we can compute (I − P + eπ). In this paper, we derive a generalization version of the fundamental matrix as (I − P + er), where r can be any given row vector satisfying re 6= 0. With this generalized fundamental matrix, we can compute g = (I−P +er)f . The steady state distribution is computed as π = r(I−P+er). The Q-factors at every state-action pair can also be computed in a similar way. These formulas may give some insights on further understanding how to efficiently compute or estimate the values of g, π, and Q-factors in Markov systems, which are fundamental quantities for the performance optimization of Markov systems.